Google And AI Chatbots Are Rewriting NHS Health Advice But No One's Checking Their Work

Dangerous advice regarding Cancer, Mental Health, And Women's Health

 

A growing accountability gap is emerging in UK healthcare as AI systems increasingly stand between NHS guidance and the patients who rely on it, according to a new analysis of the issue.

The problem
NHS.uk was built for people to browse directly — with content passing through clinical safety reviews, legal checks, and plain-English standards before publication. But when AI chatbots and search tools scrape that same content and generate their own summaries for patients, none of that oversight travels with it.

The consequences are already surfacing. A Guardian investigation in January 2026 found that Google's AI Overviews — seen by roughly 2 billion people a month — were producing false or misleading health information on topics including cancer, liver disease, women's health, and mental health. Experts reportedly flagged some of the guidance on psychosis and eating disorders as dangerous. A follow-up investigation found safety disclaimers were being visually buried beneath expanded content. Separately, the mental health charity Mind said its researchers found an AI Overview telling a tester that starvation was healthy, prompting the charity to launch what it describes as the first global year-long inquiry into AI and mental health.

England's chief medical officer reportedly told GPs in March 2026 that doctors are regularly having to correct inaccurate AI-sourced information patients bring to appointments — a burden not yet tracked in any formal safety system. A February 2026 Oxford-led study in Nature Medicine found people using AI chatbots to assess symptoms often misjudged how urgent their condition was.

Why now
Infrastructure changes are accelerating the trend. Cloudflare recently altered how it formats error messages to be more machine-readable rather than human-readable, reflecting a broader shift toward AI "agents" mediating web content rather than people browsing it directly. Cloudflare data published in mid-2025 showed OpenAI's crawler traffic share among AI bots jumping sharply over a year, with far more pages crawled than users ever referred back to source sites. UK regulators estimate roughly one in ten people already use AI chatbots for health advice.

The governance gap
NHS trust boards are used to overseeing two domains: clinical practice and digital systems. Analysts argue a third layer has emerged that nobody governs — the AI intermediaries sitting between NHS content and patients, operated by companies with no clinical accountability obligations. A professional NHS-linked medicines network has already warned that tools like ChatGPT, Gemini, and Copilot should never be a sole source for medicines information, though patients often treat them that way.

A July 2025 peer-reviewed study reportedly found NHS trusts' AI-related incident logs aren't linked into the official Serious Incident Framework, making it difficult to trace harm back to AI involvement. Industry bodies including the Professional Record Standards Body and NHS Confederation have called for mandated standards, transparency on training data, and clearer editorial accountability for AI-generated health content.

What's being proposed
Recommendations include having NHS boards map which AI systems are drawing on their published content and at what scale — data increasingly available through tools like Cloudflare Radar — and pushing the issue up to national bodies such as NHS England, the MHRA, and the Information Commissioner's Office, since no single trust can regulate global AI platforms alone. The MHRA's ongoing national commission on AI in healthcare has so far focused mainly on AI as a clinical tool rather than this consumer-facing layer.

A pattern of governance arriving too late

Commentators point to a consistent, uncomfortable pattern in NHS history: governance tends to tighten only after something has already gone wrong, not before.

The Bristol case is the starting point. Between the late 1980s and mid-1990s, mortality rates for paediatric heart surgery at Bristol Royal Infirmary were significantly higher than at comparable units, but the pattern went unaddressed for years.

The public inquiry that followed, reporting in 2001, reshaped clinical governance across the NHS — introducing systematic outcome monitoring, revalidation of clinical staff, and a much stronger expectation that poor performance be identified and acted on rather than quietly tolerated. The infrastructure for spotting harm existed in the data long before the system acted on it.

The Caldicott reviews followed a similar arc in a different domain. The first Caldicott Report, in 1997, was commissioned after growing concern that patient-identifiable information was moving around the NHS and to third parties without consistent controls or clear accountability for who could see what. It established the role of Caldicott Guardians and a set of principles now standard across the health service. Subsequent reviews, including one in 2013, extended those principles as new risks — such as large-scale data sharing — emerged. Each iteration responded to a gap that had already caused, or nearly caused, harm.

WannaCry is the most recent and most vivid example. The 2017 ransomware attack disrupted roughly a third of NHS trusts in England, cancelling thousands of appointments and operations, largely because trusts were running outdated software with known, unpatched vulnerabilities. The subsequent investigations and the Department of Health's response drove a significant tightening of NHS cyber governance, including new resilience standards and mandatory reporting. The vulnerability itself had been flagged by security researchers well before the attack occurred.

The argument being made is that each of these cases follows the same structure: a structural gap is visible in advance, but there is no established mechanism for naming it and acting on it until an incident makes the cost of inaction undeniable. The suggestion is that the AI intermediary layer — chatbots and AI-generated summaries sitting between NHS content and patients — is currently in that "visible but unnamed" phase, and that boards have an opportunity to break the pattern by treating it as a governance priority now, rather than waiting for a Bristol-, Caldicott-, or WannaCry-scale event involving AI-driven misinformation to force the issue.

 

Source: The Guardian / HSJ / Burns&Wilcox